The broad availability of generative AI offers new opportunities to support various work domains, including agile software development. Agile epics are a key artifact for product managers to communicate requirements to stakeholders. However, in practice, they are often poorly defined, leading to churn, delivery delays, and cost overruns. In this industry case study, we investigate opportunities for large language models (LLMs) to evaluate agile epic quality in a global company. Results from a user study with 17 product managers indicate how LLM evaluations could be integrated into their work practices, including perceived values and usage in improving their epics. High levels of satisfaction indicate that agile epics are a new, viable application of AI evaluations. However, our findings also outline challenges, limitations, and adoption barriers that can inform both practitioners and researchers on the integration of such evaluations into future agile work practices.

Peter (not his real name) is a product manager at a data storage company, which offers its people the Google Gemini AI chatbot. External AI tools are banned but Peter uses ChatGPT through search tool Kagi. He finds the biggest benefit of AI comes from challenging his thinking when he asks the chatbot to respond to his plans from different customer perspectives. "The AI is not so much giving you answers, as giving you a sparring partner," he says. "As a product manager, you have a lot of responsibility and don't have a lot of good outlets to discuss strategy openly. These tools allow that in an unfettered and unlimited capacity."

We explore the potential for productive team-based collaboration between humans and Artificial Intelligence (AI) by presenting and conducting initial tests with a general framework that enables multiple human and AI agents to work together as peers. ChatCollab's novel architecture allows agents - human or AI - to join collaborations in any role, autonomously engage in tasks and communication within Slack, and remain agnostic to whether their collaborators are human or AI. Using software engineering as a case study, we find that our AI agents successfully identify their roles and responsibilities, coordinate with other agents, and await requested inputs or deliverables before proceeding. In relation to three prior multi-agent AI systems for software development, we find ChatCollab AI agents produce comparable or better software in an interactive game development task. We also propose an automated method for analyzing collaboration dynamics that effectively identifies behavioral characteristics of agents with distinct roles, allowing us to quantitatively compare collaboration dynamics in a range of experimental conditions. For example, in comparing ChatCollab AI agents, we find that an AI CEO agent generally provides suggestions 2-4 times more often than an AI product manager or AI developer, suggesting agents within ChatCollab can meaningfully adopt differentiated collaborative roles. Our code and data can be found at: https://github.com/ChatCollab.

Synthetic data generation is increasingly recognized as a crucial solution to address data related challenges such as scarcity, bias, and privacy concerns. As synthetic data proliferates, the need for a robust evaluation framework to select a synthetic data generator becomes more pressing given the variety of options available. In this research study, we investigate two primary questions: 1) How can we select the most suitable synthetic data generator from a set of options for a specific purpose? 2) How can we make the selection process more transparent, accountable, and auditable? To address these questions, we introduce a novel approach in which the proposed ranking algorithm is implemented as a smart contract within a permissioned blockchain framework called Sawtooth. Through comprehensive experiments and comparisons with state-of-the-art baseline ranking solutions, our framework demonstrates its effectiveness in providing nuanced rankings that consider both desirable and undesirable properties. Furthermore, our framework serves as a valuable tool for selecting the optimal synthetic data generators for specific needs while ensuring compliance with data protection principles.

In long document controllable summarization, where labeled data is scarce, pretrained models struggle to adapt to the task and effectively respond to user queries. In this paper, we introduce Socratic pretraining, a question-driven, unsupervised pretraining objective specifically designed to improve controllability in summarization tasks. By training a model to generate and answer relevant questions in a given context, Socratic pretraining enables the model to more effectively adhere to user-provided queries and identify relevant content to be summarized. We demonstrate the effectiveness of this approach through extensive experimentation on two summarization domains, short stories and dialogue, and multiple control strategies: keywords, questions, and factoid QA pairs. Our pretraining method relies only on unlabeled documents and a question generation system and outperforms pre-finetuning approaches that use additional supervised data. Furthermore, our results show that Socratic pretraining cuts task-specific labeled data requirements in half, is more faithful to user-provided queries, and achieves state-of-the-art performance on QMSum and SQuALITY.

Generative Artificial Intelligence (GAI) has made outstanding strides in recent years, with a good-sized impact on software product management. Drawing on pertinent articles from 2016 to 2023, this systematic literature evaluation reveals generative AI's potential applications, benefits, and constraints in this area. The study shows that technology can assist in idea generation, market research, customer insights, product requirements engineering, and product development. It can help reduce development time and costs through automatic code generation, customer feedback analysis, and more. However, the technology's accuracy, reliability, and ethical consideration persist. Ultimately, generative AI's practical application can significantly improve software product management activities, leading to more efficient use of resources, better product outcomes, and improved end-user experiences.

In the 1990s, business process reengineering was all the rage: Companies used budding technologies such as enterprise resource planning (ERP) systems and the internet to enact radical changes to broad, end-to-end business processes. Buoyed by reengineering's academic and consulting proponents, companies anticipated transformative changes to broad processes like order-to-cash and conception to commercialization of new products. But while technology did bring major updates, implementations often failed to live up to the sky-high expectations. For example, large-scale ERP systems like SAP or Oracle provided a useful IT backbone to exchange data, yet also created very rigid processes that were hard to change past the IT implementation. Since then, process management typically involved only incremental change to local processes -- Lean and Six Sigma for repetitive processes, and Agile Lean Startup methods for development -- all without any assistance from technology.

As a member of the Machine Learning team, you'll work with both methods researchers and small molecule designers to imagine and design user experiences to leverage machine learning methods. This position offers the opportunity to influence Schrödinger's business direction and scientific functionality by bridging gaps between technical, scientific, and commercial realms.

Skydio is the leading US drone company and the world leader in autonomous flight, the key technology for the future of drones and aerial transportation. The Skydio team combines deep expertise in artificial intelligence, best-in-class hardware and software product development, and operational excellence to empower a broader, more diverse audience of drone users - from action sports enthusiasts to first responders to insurance claims adjusters. About the role: As a Product Manager for Autonomous Drone Hardware, you will work closely with our hardware and embedded software engineering teams to help define and shape Skydio's next generation drones and controllers. You'll ensure that we are building and delivering the right products and features for our entire range of customers. How you'll make an impact: Compensation Range: The annual base salary range for this position is $144,500 - 182,750*.

During economic downturns, businesses resort to muscle memory and do what they've done before. That often means budget cuts--and the deepest cuts commonly target technology investment and people. This time, however, things already feel very different. Businesses increasingly see their tech talent as a hard-won strategic investment, which they are reluctant to lose. New McKinsey & Company research discovered that 55 percent of 1,100 companies surveyed globally have found it challenging to hire key data and tech roles, such as data and software engineers, data architects, machine learning engineers, and data scientists.